ABSTRACT
The past 3 years of work in NLP have been characterized by the
development and deployment of ever larger language models, es-
pecially for English. BERT, its variants, GPT-2/3, and others, most
recently Switch-C, have pushed the boundaries of the possible both
through architectural innovations and through sheer size. Using
these pretrained models and the methodology of fine-tuning them
for specific tasks, researchers have extended the state of the art
on a wide array of tasks as measured by leaderboards on specific
benchmarks for English. In this paper, we take a step back and ask:
How big is too big? What are the possible risks associated with this
technology and what paths are available for mitigating those risks?
We provide recommendations including weighing the environmen-
tal and financial costs first, investing resources into curating and
carefully documenting datasets rather than ingesting everything on
the web, carrying out pre-development exercises evaluating how
the planned approach fits into research and development goals and
supports stakeholder values, and encouraging research directions
beyond ever larger language models.

Emily M. Bender
University of Washington
Seattle, WA, USA

Timnit Gebru
Black in AI
Palo Alto, CA, USA

Angelina McMillan-Major
University of Washington
Seattle, WA, USA

Shmargaret Shmitchell
The Aether

ACM Reference Format:
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmar-
garet Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? . In Conference on Fairness, Accountability, and Transparency (FAccT ’21), March 3–10, 2021, Virtual Event, Canada. ACM, New York, NY, USA, 14 pages. https://doi.org/10.1145/3442188.3445922